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Abstract 

Background: The CanMEDS roles and competencies are being used as the framework to support the development 
of the Manitoba Practice Assessment Program (MPAP) designed to assess the competence of physicians practicing 
with a conditional license. Establishing the link between clinical practice and assessment of performance is critical 
in the development of the MPAP. A first step in establishing this link is to identify activities performed in actual 
clinical practice as well as the importance of those activities. 

Methods: A descriptive survey design was used to answer the research questions: (1) How do physicians rate the 
complexity, frequency, and criticality of CanMEDS roles? (2) What is the distribution of perceived importance scores 
for the CanMEDS roles? Two online surveys, one specific to family practice physicians, and one specific to 
specialists, were emailed to a sample of Canadian physicians. 

Results: Overall perceived importance scores were calculated for each of the CanMEDS roles. It appears that each 
role is considered to be at least moderately important. The Medical Expert role was ranked as the most important, 
followed by the roles of Communicator, Professional, Collaborator, Scholar, Manager, and Health Advocate. There 
were no significant differences in overall CanMEDS perceived importance scores between family practice physicians 
and specialists (A/ = 88). 

Conclusions: Given that each of the CanMEDS roles is considered at least moderately important, a variety of 
assessment tools are needed to evaluate competencies across the entire spectrum of roles. The results underscore 
the importance of incorporating a multifaceted approach when developing a practice assessment program. 
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Background 

The University of Manitoba, Faculty of Medicine, Div- 
ision of Continuing Professional Development is 
embarking on the development of an assessment pro- 
gram for practicing physicians called the Manitoba Prac- 
tice Assessment Program (MPAP). The MPAP is being 
developed in response to a need identified by the College 
of Physicians and Surgeons of Manitoba (CPSM). Within 
the Province of Manitoba, physicians commonly enter 
practice on a conditional license. A conditional license 
allows physicians who have completed post-graduate 
training, but have not yet achieved their Canadian certi- 
fication through The College of Family Physicians of 
Canada (CFPC) or the Royal College of Physicians and 
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Surgeons of Canada (RCPSC), to enter supervised prac- 
tice within a defined time frame during which they are 
expected to complete the appropriate certification exam. 
There are a significant number of physicians who are in 
established practices and have not yet achieved final cer- 
tification for a variety of reasons. The CPSM requested 
that the University of Manitoba, Faculty of Medicine de- 
velop a process to assess such physicians in their prac- 
tice settings to enable the CPSM to make a decision 
about granting registration to these candidates. 

The framework for the development of the MPAP is 
the Canadian Medical Education Directions for Specia- 
lists (CanMEDS) physician competency framework. The 
establishment of the CanMEDS framework is rooted in 
medical education with the framework originally devel- 
oped to prepare physicians to thrive in our ever- 
changing healthcare environment [1]. Since 1993, the 
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framework has evolved through four major phases with 
the current 2005 version highlighting the seven core 
roles of a physician: Medical Expert, Communicator, 
Collaborator, Manager, Health Advocate, Scholar, and 
Professional [1]. Key and enabling competencies have 
been developed for each of the core CanMEDS roles. A 
trademarked diagram, known as either the CanMEDS 
cloverleaf, daisy, or flower, was produced by the RCPSC 
to visually illustrate the core roles and interconnections 
among the roles [1]. Essentially, the Medical Expert is in 
the middle of the daisy representing the central role, 
with each of the remaining roles represented by an over- 
lapping petal that is equal in size to one another. 

The CFPC [2] adopted the CanMEDS framework 
developed by the RCPSC [3]; however, it was customized 
to reflect the specific competencies of family practice 
physicians specifically in the Medical Expert role with 
the addition of two competencies, and the Collaborator 
and Professional roles with one competency being added 
to each role. Other than these additions and a few minor 
wording changes, the two frameworks are essentially the 
same. The RCPSC specialties and sub-specialties have 
also adapted the CanMEDS framework to reflect their 
specific practices [4]. 

The CanMEDS framework has been incorporated into 
undergraduate and postgraduate educational programs 
both within Canada and internationally [5-11]. Suc- 
cesses can be attributed to a detailed implementation 
strategy that has focused on four key areas: (1) standards 
for curriculum, teaching, and assessment; (2) faculty de- 
velopment; (3) research and development resources; and 
(4) outreach and communications [12]. Gaps, however, 
have been noted between the ideal delivery of medical 
education that incorporates all of the core CanMEDS 
roles into the curriculum as opposed to a focus on the 
role of Medical Expert [13,14]. Frank and Danoff [12] ac- 
knowledge that resistance to change, faculty overload, 
and scarce financial resources are challenges that limit 
the adoption of the CanMEDS framework. 

The concept of using the CanMEDS framework as a 
foundation for an assessment program for practicing phy- 
sicians is relatively novel as the focus has been on under- 
graduate, postgraduate, and continuing medical education. 
Researchers have started to examine the importance rank- 
ings of the CanMEDS competencies; however, the context 
of the research focused on the applicability of the 
CanMEDS roles and competencies for international physi- 
cians [8,9]. To date, no studies have been located in which 
researchers specifically examined the activity weight 
scores of roles by asking practicing Canadian physicians to 
rate the criticality or consequence of performing compe- 
tencies key to their practice incorrectly or not at all, or by 
rating the frequency in which they perform these key 
competencies on an everyday basis. 



It is important that a credentialing or licensing pro- 
gram be able to provide empirical evidence of the link 
between the knowledge, skills, and attitudes tested in an 
examination and the activities performed in practice 
[15]. Although the MPAP is not a credentialing or li- 
censing program, the MPAP assessment process is con- 
sidered a high-stakes assessment given that the outcome 
of the assessment will contribute to the CPSM s decision 
regarding registration. Establishing the link between 
clinical practice and assessment of performance is crit- 
ical in the development of the MPAP, and a first step in 
establishing this link is to clearly identify activities per- 
formed in actual clinical practice, for assessment tools 
need to include all roles and competencies and not just 
focus on the Medical Expert role. In addition, the 
weighting of the various roles needs to be determined 
and taken into consideration when developing the as- 
sessment tools and determining the final pass/fail out- 
come of an assessment. 

The goal of the study was to determine the importance 
of physicians roles and competencies as perceived by a 
sample of Canadian physicians. The following questions 
guided the research: 

1. How do physicians rate the complexity, frequency, 
and criticality of CanMEDS roles? 

2. What is the distribution of perceived importance 
scores for the CanMEDS roles? 

Methods 

A descriptive survey design was used to answer the re- 
search questions and achieve the study goal by asking 
practicing physicians to rate competencies identified as 
key to their practice. Ultimately, the results will be taken 
into consideration in the development of the assessment 
tools and processes that will be used by the MPAP in 
assessing the competence of practicing physicians. 

Procedure 

After receiving ethical approval from the University of 
Manitoba Bannatyne Campus Research Ethics Boards 
(File Number: H2010:357) to conduct the study, a list of 
Canadian physicians residing in British Columbia, Alberta, 
Saskatchewan, Manitoba, North West Territories, Yukon 
Territory, and Nunavut was generated by the principal 
investigator from the online database called, Scott's Direc- 
tories: Canadian Medical Directory via a departmental 
username and password. Information obtained from the 
directory included the physicians first and last name, 
province of residence, and email address. Physicians that 
met the inclusion criteria, which was a valid email address, 
were included. 

Due to the large participant list, and to avoid the re- 
cruitment notice being identified as unsolicited mail, an 
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email distribution service, namely, Constant Contact, 
was used to send the electronic recruitment notice. 
Included in the recruitment notice was general partici- 
pant information, an electronic copy of the consent 
form, and links to two online surveys. The principal in- 
vestigator designed and sent the recruitment email to 
potential participants. If an email was returned undeliv- 
ered, no attempt was made to resend the email. Physi- 
cians were solely responsible for reading the participant 
information and consent form and determining for 
themselves whether to participate in the study. Partici- 
pants were not required to sign and return the informed 
consent, and completion of the online survey constituted 
consent to participate. In addition, a statement of con- 
sent was included on the introductory page of the online 
survey. Participants were provided with contact informa- 
tion for all of the investigators as well as the university 
research ethics board coordinator if they had any ques- 
tions, issues, or concerns. 



After deciding to participate, physicians completed 
one of two researcher developed online surveys access- 
ible via an encrypted subscription of Survey Monkey. 
Physicians with a clinical background in Family Practice 
were asked to complete the survey entitled, MPAP: 
CanMEDS Competencies for Family Practice. Physicians 
with a Specialty Practice clinical background were asked 
to complete the survey entitled, MPAP: CanMEDS Com- 
petencies for Specialty Practice. A period of two months 
was given to complete the survey. One month after 
sending out the initial recruitment email, the principal 
investigator sent a follow-up reminder email to all parti- 
cipants. Since it was not known who completed the sur- 
veys, all participants received the follow-up reminder 
email. Constant Contact has an option for opting out of 
the email service, and those physicians that opted out of 
receiving any further emails after the initial recruitment 
email did not receive the follow-up email. Approxi- 
mately one month after access to the surveys closed, 



MPAP: CanMEDS Competencies for Family Practice 


11. CanMEDS Role: Communicator 2.0 





The three questions on this page refer to the following competency: 

KEY COMPETENCY 

ACCURATELY ELICIT AND SYNTHESIZE INFORMATION FROM. AND PERSPECTIVES OF. PATIENTS AND FAMILIES. COLLEAGUES AND 
OTHER PROFESSIONALS 

Enabling Competencies: 

'Gather reformation about a dsease. but also about a pat ent s be-: efs. concerns, expectations and illness experience 
'Explore the patents psychosocial context 

•Seek out and synthesize nformabon from other sources, such as a patient s famfy. caregivers and other profess onals 
•Conduct an merview with multiple participants to gather information about factors affecting the patent 



1. C2.0 What level of knowledge or skill is required to perform this competency? 



Complexity 



Knowledge 

O 



2 

O 



2 

o 



2. C2.0 How often do you perform this competency? 

1 = Rarefy 2 3 

Frequency O O O 



O 

4 

o 



5 = High Level of 
Knowlecge 

O 



5 = All the time 

O 



3. C2.0 What would be the consequences of performing this competency incorrectly or not 
at all? 



Critical ity 



1 = No consequence 

O 



o 



o 



o 



5 = Extreme 
consequence 



o 



Figure 1 Example of survey questions. 
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overall results were sent to all participants who received 
the initial recruitment email and had not opted out of 
receiving future emails. 

Surveys 

The surveys were developed by the investigators apply- 
ing the work of Raymond [15] and using the actual 
wording of CanMEDS roles, key competencies, and en- 
abling competencies from the RCPSC [3] and the CFPC 
[2]. Based on a table of scales for rating practice activ- 
ities and practice-related responsibilities from a variety 
of studies [15], the following questions were developed 
to measure complexity, frequency, and criticality of each 
of the key competencies: (1) Complexity: What level of 
knowledge or skill is required to perform this compe- 
tency? (2) Frequency: How often do you perform this 
competency? (3) Criticality: What would be the conse- 
quences of performing this competency incorrectly or 
not at all? A rating scale of 1 to 5 was used for all three 
questions (see Figure 1). Family practice physicians were 
asked to score 31 key CFPC competencies, and specialty 
practice physicians were asked to score 28 key RCPSC 
competencies based on the frequency, complexity, and 
criticality of the competency in their own practice. 
Included in the survey was a demographic section where 
physicians described their practices as active or inactive, 
and urban, rural, or northern/remote. Time to complete 
the survey was approximately 30 minutes. 

Data Analysis 

An overall index of perceived importance for each com- 
petency (i) within its respective CanMEDS role was cal- 
culated using a multiplicative model 

= FtQ 

where I t = overall importance, F t = frequency, and Q = 
criticality [15-17]. A common limitation of the multi- 
plicative model occurs when the range of criticality rat- 
ings is relatively narrow compared to the variability of 
frequency ratings leading to biased ratings of overall im- 
portance. In this instance, it is necessary to transform 
the criticality ratings to equalize their contributions in 
calculating perceived overall importance [16]. However, 
in our data, because these descriptives for each compe- 
tency were not markedly different, the variables were 
not transformed. Moreover, the pattern of ratings 
derived from the multiplicative model are comparable to 
other complex methods of combining frequency and 
criticality ratings [17]. An index of practice importance 
for each role was obtained by calculating the mean per- 
ceived overall importance of the competencies within 
the role [15]. 



Results and discussion 

A total of 3,294 participants meeting the study criteria 
received the initial recruitment email. Five hundred and 
sixty two emails were returned undelivered, and a total 
of 27 participants opted out of receiving emails from the 
research team over the course of the study. Thirty five 
participants accessed and started to complete the MPAP: 
CanMEDS Competencies for Family Practice survey and 
92 participants accessed and started to complete the 
MPAP: CanMEDS Competencies for Specialty Practice 
survey. Scores from those participants who did not 
complete the majority of survey questions were deleted 
prior to analysis of data. The final sample (AT =88) con- 
sisted of 23 participants completing the family practice 
survey and 65 completing the specialty practice survey. 
Although the 3% response rate fell well short of an initial 
expected minimum of 10%, based on the work of Tan- 
nenbaum and Wesley [18] who concluded that subject 
matter experts (N = 7) and field respondents (N = 329 
and 423) provided similar ratings of importance for 
knowledge and ability domains in job analysis surveys, 
the results do provide promising preliminary data upon 
which further studies can build. Numerous factors may 
have contributed to the low response rate including the 
length of the survey that included 95 questions for fam- 
ily practice physicians and 86 questions for specialty 
practice physicians. Physicians may not have had the 
time to complete such a lengthy survey without any 
compensation or tangible incentive for participation. 
The fact that the survey was an online survey may have 
contributed to the low response rate, for according to 
the Constant Contact statistics, 517 potential respon- 
dents did not even open the email request to participate, 
and as noted, 562 emails were not delivered due to in- 
valid email addresses. In hindsight, using a marketing 
service database to identify potential research partici- 
pants (i.e., Scotts Directories: Canadian Medical 
Directory) may not have been optimal, for those physi- 
cians on the list likely get numerous unwanted emails 
and the request to participant may have been perceived 
as junk mail. Strategies to increase the response rate 
may include targeting specific physician groups through 
physician organizations and special interest groups, as 
well as distributing hardcopy surveys as opposed to 
using an online survey; however, hardcopy distribution 
will add to the cost of the survey in terms of the mail- 
out itself and data entry costs. A follow-up focus group 
to review the results may also be helpful in the validation 
process. Despite the low response rate, the researchers are 
unaware of any other work being done to attempt to 
weight the CanMEDS roles; therefore, the results are an 
important step in the continuous validation of the 
CanMEDS framework. Of those physicians that partici- 
pated, 70 claimed to be in active practice, three referred to 
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themselves as inactive practice, and 15 did not declare 
their status. Location of participants' practice was noted as 
urban (n = 52), rural (n = 3), or northern/remote (n = 1), 
while 7 noted a combination of locations, and 25 did not 
note a practice location. 

The mean ratings of complexity, frequency, criticality, 
and perceived overall importance for the competencies 
in each role are shown in Table 1. For complexity, the 
roles were fairly consistent, ranging from 3.81 (Profes- 
sional) to 4.37 (Medical Expert) indicating that most 
competencies within each role required a moderate to 
high level of skill. The mean perceived frequencies were 
slightly more varied, ranging from 3.32 (Health Advo- 
cate) to 4.22 (Medical Expert). The Medical Expert role 
had the highest mean rating of Criticality at 4.09 com- 
pared to the least critical role, namely Health Advocate 
at 3.41. As mentioned earlier, perceived overall import- 
ance represented the product of the mean scores of fre- 
quency and criticality. As such, the lowest and highest 
possible mean scores of perceived overall importance 
range from 1 to 25 respectively. In sum, it appears that 
each role is considered to be at least moderately import- 
ant ranging from 11.93 (Health Advocate) to 17.71 
(Medical Expert). No significant differences in mean per- 
ceived importance scores were found in any of the 
CanMEDS roles between those participants that com- 
pleted the family practice survey and those that com- 
pleted the specialty practice survey (see Table 2). 

The finding that there was no significant difference in 
perceived overall importance scores for CanMEDS roles 
between participants that completed the family practice 
survey and the specialty practice survey is an important 
result as the research team moves forward in the devel- 
opment of a physician practice assessment program. The 
finding supports the notion that one set of comprehen- 
sive practice assessment tools can be developed to assess 
both family practice and specialty practice physicians 
using the CanMEDS roles and competencies as the 
underlying framework; however, additional specialty spe- 
cific competencies will need to be included in specialty 
specific assessment tools. 



Table 2 Comparison of mean importance scores for 
family practice and specialty physicians 



CanMEDS Role 




n 


M 


SO 


t 


P 


Medical Expert 


Family 
Specialty 


23 65 


1 7.07 1 7.93 


3.69 4.21 


-.87 


.38 


Communicator 


Family 
Specialty 


23 65 


17.29 16.11 


2.81 4.94 


1.08 


.28 


Collaborator 


Family 
Specialty 


23 63 


15.04 14.40 


3.81 5.16 


.55 


.59 


Manager 


Family 
Specialty 


22 63 


12.52 12.79 


4.35 4.79 


-.24 


.82 


Health Advocate 


Family 
Specialty 


22 62 


12.30 11.08 


3.95 4.90 


.43 


.67 


Scholar 


Family 
Specialty 


22 62 


12.20 13.69 


4.13 4.93 


-1.27 


.21 


Professional 


Family 
Specialty 


22 61 


16.22 14.98 


3.78 4.73 


1.11 


.27 



The findings support the work of Frank [1] in that the 
Medical Expert role, that received the highest perceived 
overall importance score, is a central integrative role em- 
bodied by competent physicians. However, when exam- 
ining the holistic practice of physicians, the roles of 
Communicator, Collaborator, Health Advocate, Manager, 
Scholar, and Professional play an important part in sup- 
porting the role of Medical Expert. The perceived im- 
portance score of the Collaborator role was somewhat 
surprising, especially given the focus on interprofessional 
practice and the need to engage with other physicians 
and health professionals on a daily basis in order to pro- 
vide quality patient care. Specific reference to communi- 
ties and the population at large, in questions pertaining 
to the Health Advocate role, may have contributed to 
the low ranking by specialty physicians in particular, for 
time constraints in their practices may not allow for 
health advocacy at this more global level. Possibly a lar- 
ger sample size would have yielded different results. 

The importance scores of each of the CanMEDS roles 
will be beneficial in determining a physicians overall 
competence as assessed in a practice assessment 



Table 1 Descriptives for CanMEDS Roles 



CanMEDS Role 


n 


Complexity M (SO) 


Frequency M (SO) 


Criticality M (SO) 


Overall Importance M (SO) 


Medical Expert 


88 


4.37 (0.47) 


4.22 (0.71) 


4.09 (0.54) 


17.71 (4.08) 


Communicator 


88 


4.03 (0.71) 


4.20 (0.79) 


3.82 (0.66) 


16.42 (4.50) 


Collaborator 


86 


3.99 (0.87) 


3.86 (0.85) 


3.69 (0.76) 


14.56 (4.82) 


Manager 


85 


3.97 (0.76) 


3.57 (0.88) 


3.42 (0.73) 


12.72 (4.65) 


Health Advocate 


84 


3.99 (0.77) 


3.32 (0.94) 


3.41 (0.73) 


11.93 (4.66) 


Scholar 


84 


4.20 (0.72) 


3.67 (0.92) 


3.51 (0.70) 


13.30 (4.75) 


Professional 


83 


3.81 (0.86) 


3.86 (0.75) 


3.86 (0.72) 


15.31 (4.51) 
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program. It is expected that the relative importance of 
each of the roles will be taken into consideration when 
determining an overall pass or fail grade. Beyond assess- 
ment, importance and complexity scores can be used as 
a reference when developing undergraduate, postgradu- 
ate, and continuing medical education curricula. 

Interestingly, Frank [1] notes that based on history and 
tradition, the CanMEDS roles are ranked as follows: 
Medical Expert, Communicator, Collaborator, Manager, 
Health Advocate, Scholar, and Professional. Given the 
relative importance ranking of roles from this study, and 
the fact that Whitehead, Austin, and Hodges [19] ques- 
tion the historical ranking of the roles, it may be that 
the ordering of the CanMEDS roles will need to change 
with further validation of results? 

As the CanMEDS roles and competencies continue to 
be adopted in all facets of medicine, it is important that 
the framework continues to be validated. The small sam- 
ple size limits the generalizability of the findings and 
researchers need to continue to examine perceived im- 
portance scores of the CanMEDS roles with larger sam- 
ples and with specific specialties and sub-specialties. 
Many specialty groups certified by the RCPSC, for ex- 
ample, have adapted the CanMEDS roles and competen- 
cies to their own practices [4], and similar to the finding 
of Ringsted et al. [9] that discovered differences in im- 
portance ratings among specialties, it would be expected 
that the importance scores from one specialty to the 
other may be different. For example, the importance 
score for the Communicator role would be expected to 
be different for a psychiatrist compared to a pathologist. 
The non-differentiation of sub-specialties is a limitation 
of this study. 

Conclusions 

Given that each of the CanMEDS roles is considered at 
least moderately important, it is assumed that a variety 
of assessment tools are needed to evaluate competencies 
across the entire spectrum as opposed to focusing on 
the role of Medical Expert. Overall, the results under- 
score the importance of incorporating a multifaceted ap- 
proach when developing a practice assessment program. 
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